Skip to content

Stateful encoders + documentation refresh - #66

Merged
cvigilv merged 18 commits into
mainfrom
docs-encoders-refresh
Jul 29, 2026
Merged

Stateful encoders + documentation refresh#66
cvigilv merged 18 commits into
mainfrom
docs-encoders-refresh

Conversation

@MichielStock

Copy link
Copy Markdown
Collaborator

Brings the stateful encoders and the tutorial rewrite together, ports the tutorials to the
current API, and gives the encoder layer its first documentation.

The two strands are combined because they are not separable: the encoder work deletes
level/encodelevel/decodelevel/convertlevel, and the tutorials use all four. Merging
either side alone leaves main with docs that do not build.

Replaces

PR Branch Why
#64 encoders → main merged into this branch
#65 encodersdocs-new-tutorials same integration, done here against 38-integration
#63 38-integration → main this branch is 38-integration plus the encoder merge
#38 cvigilv:docs-new-tutorials its intro tutorial is merged in here (credit to @cvigilv)

#28 (docs_types) still touches src/vectors.jl, which no longer exists — it looks obsolete,
but nothing here deliberately covers it, so that is a separate call.

What is in it

Stateful encoders (from #64): LevelEncoder for scalars and RandomProjection for feature
vectors, both under AbstractEncoder{HV}, with encode/decode and rethreshold. This
replaces the old level/encodelevel/decodelevel/convertlevel family, and along the way
retires the bug where convertlevel built its encoder and decoder over two different random
ladders.

Tutorials brought to the current API:

  • iris.jl used all four removed functions, and defined local helpers named encode and
    decode that now collide with the package's exports. Ported to a single LevelEncoder;
    helpers renamed. The prose got simpler: it used to explain by hand that encoder and decoder
    must share a ladder, which is exactly what LevelEncoder now enforces by construction.
  • api.md listed the four removed functions in an @docs block (a hard build error).
    Restructured into Combinators / Encoders matching the layer taxonomy, which also gives
    the stateful encoders their first API-page entry.
  • Introduction tutorial: best of both drafts — this branch's heading hierarchy and prose, plus
    the concept/kitchen-analogy table and the closing wrap-up from docs: new tutorials #38.

New documentation:

  • Encoding data — the encoder layer's first coverage: token path, the KMer vs NGram
    distinction (each shown equivalent to the combinators it is built from), Sequence/
    BagOfSymbols, and the extension point. Headline example is language recognition from
    character k-mers: seven Wikipedia snippets, one encode call each, and the similarity ranking
    recovers real linguistic structure (English/Simple-English, Dutch/West-Flemish, then the
    Romance and Germanic pairs) with cross-family pairs at the bottom. This is the payoff for New k-mers encoder #53.
  • ColoursRandomProjection end to end: encoding, blending, decode as codebook clean-up
    (and why it cannot be inversion), an associative memory recovering each category's average
    colour, the same with two-thirds of every observation replaced by noise, the tuning knobs, and
    a plain-Julia perceptron sharpening a boundary (0.85 → 0.96). The kernel connection is
    verified numerically in the page: FHRR + Gaussian matches exp(-β²d²/2) to three decimals.
  • Iris act two — the same workflow via one RandomProjection call, scoring comparably
    (0.96 vs 0.93), which sets up the real trade-off between an interpretable record encoding and
    a one-shot projection.
  • Landing page rewritten (it pointed at the wrong repository and its @contents referenced a
    non-existent examples.md, so the manual section rendered empty); developer guide added to the
    nav; Remotes.GitHub silences the navbar warning.

Two small API additions, both from review questions on the docs:

  • sum(hvs) now bundles the whole collection. Bundling is m-way and therefore not
    associative for BinaryHV/BipolarHV/RealHV/FHRR. Chained x + y + z was already correct
    (Julia parses a + chain as one variadic call), but sum, reduce(+, …) and (x + y) + z
    fold pairwise and bundle a bundle. Measured on five hypervectors, the folded result scores
    0.08/0.05/0.14/0.26/0.50 against its inputs where a true bundle scores ~0.37 for all five —
    the defining property of bundling is silently lost. sum now does the right thing; the
    remaining pairwise spellings are documented as wrong.
  • similaritymetric(HV) and chancesimilarity(HV). A similarity score is uninterpretable
    without its metric: 0.35 means unrelated under Jaccard (BinaryHV, GradedHV) and
    clearly related under cosine. chancesimilarity returns the score of an unrelated pair
    (0.0 cosine, 1/3 Jaccard) and is locked by a test asserting the documented baseline matches
    what random hypervectors actually score.

Breaking

  • level, encodelevel, decodelevel, convertlevel are removed — use LevelEncoder.
  • sum(hvs) now bundles all at once instead of folding pairwise (the old result was wrong).

Notes for review

  • docs/src/examples/*.md is Literate output regenerated on every build; it was tracked and made
    up a third of this diff as machine-generated duplicates of the .jl sources. Now untracked and
    gitignored — verified the docs build from a clean state with no .md present.
  • Two measured findings are written into the docs rather than the usual folklore: standardising
    features before a random projection hurts on iris (0.83 ± 0.06 vs 0.98 ± 0.03 raw, raw
    winning 29 of 30 splits) because its features already share a unit and the largest-spread ones
    are the most discriminative; and the pairwise-bundling degradation above.

Verification

866 tests, doctests, and a warning-clean docs/make.jl build all green. Every changed .jl
checked against Runic, so the style job should be clean.

cvigilv and others added 15 commits July 15, 2026 01:00
Completes the stateful AbstractEncoder pair begun with LevelEncoder:

- RandomProjection{HV, T}: fixed D×d projection matrix (gaussian /
  bipolar / sparse_ternary), per-type nonlinearities, scalar-or-vector
  ternary threshold θ with a data-driven target_sparsity constructor,
  supplied-matrix constructor, rethreshold sharing R, and
  nearest-neighbour-only decode (lossy encoding: no analytic inverse).
- phase_encode(z, β): single shared FHRR phase-encoding helper; both
  LevelEncoder's fractional power path and RandomProjection's FHRR
  nonlinearity (random Fourier features) route through it.
- Regression tests for two known sharp edges: the ternary constructor's
  positional collision (supplied matrix vs training data; the residual
  square-matrix ambiguity and silently ignored kwargs are recorded as
  TODO §2.6, fix is a follow-up rename) and BipolarHV's raw-bits /
  zero-state polarity traps, pinned as behaviour-locking tests so the
  obvious-but-wrong idioms trip red instead of shipping corruption.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resolves the src/encoding.jl conflict by dropping the level/encodelevel/
decodelevel/convertlevel family (deleted by the encoders work) while keeping
this branch's docstring reformatting for every surviving combinator.

Fallout fixed:
- api.md listed the four removed functions in an @docs block; restructured into
  Combinators / Encoders sections matching the primitives -> combinators ->
  encoders taxonomy, and added decode, AbstractEncoder, LevelEncoder,
  RandomProjection and rethreshold.
- iris.jl used all four removed functions, and also defined local helpers named
  encode and decode which now collide with the package's exported ones. Ported
  to a single LevelEncoder and renamed the helpers to encodeflower/decodeflower.
  The prose now names the concept it was already explaining by hand: a stateful
  encoder is shared state, which is what makes encode and decode consistent.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keeps this branch's heading hierarchy and refined prose, and takes two things
from the other draft: the HDC-concept/kitchen-analogy table in the opening,
which orients the reader before any code, and the closing wrap-up, which this
version lacked entirely.

Also updates the 'more encoders' admonition, which still advertised the removed
level encoders, to name the combinator/encoder split and the new encoders.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New Literate tutorial giving the encoder layer its first documentation: the
token path, the KMer/NGram distinction (with a worked equivalence to the
combinators each is built from), Sequence/BagOfSymbols, and short sections on
LevelEncoder and RandomProjection pointing at the dedicated tutorials.

The headline example is language recognition from character k-mers, ported from
the HDC workshop: seven Wikipedia snippets, one encode call each, and the
similarity ranking recovers real linguistic structure (English/Simple English,
Dutch/West-Flemish, then the Romance and Germanic pairs) with cross-family pairs
at the bottom. This is the payoff for issue #53 and the clearest demonstration
of why KMer is its own strategy.

Also documents the extension point with a working custom strategy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gives RandomProjection its first tutorial coverage, built on the workshop's
colour example:

- encoding RGB triples so perceptual closeness becomes hypervector similarity
- blending by bundling
- decode as clean-up against a codebook, with the reason it cannot be inversion
- an associative memory learning each category's average colour via
  bind/bundle/unbind, and the same thing again with two-thirds of every
  observation replaced by noise (multiple-instance learning)
- the tuning knobs: matrix flavours, theta, and rethreshold sharing the matrix
- the kernel connection: FHRR + gaussian is random Fourier features, verified
  numerically against exp(-b^2 d^2/2) in the page itself
- a plain-Julia perceptron sharpening the yellow/green boundary (0.85 -> 0.96),
  explicitly flagged as not-yet-package-API

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
index.md pointed at the wrong repository and its @contents block referenced a
non-existent examples.md, so the manual section rendered empty. Replaced with a
real landing page: what HDC is, installation, a runnable first taste, the
three-layer taxonomy as a table, and links to every tutorial.

Also adds the developer guide to the nav (it was built but unreachable) and
passes repo as Remotes.GitHub, which silences Documenter's 'unable to determine
repository root' navbar warning.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second act for the iris example: the same nearest-prototype workflow, but with
the four measurements encoded by a single RandomProjection instead of the
key-value + level construction. Scores comparably (0.96 vs 0.93) from an encoder
that needs no domain knowledge, which sets up the real trade-off: the key-value
representation can be decoded feature by feature, a random projection cannot.

Includes a measured warning about standardising. The usual advice is to
standardise before projecting, but on iris that makes things markedly worse
(0.83 +/- 0.06 vs 0.98 +/- 0.03 raw, raw winning 29 of 30 splits) because the
features already share a unit and the largest-spread ones are the most
discriminative. Softened the same blanket advice in CLAUDE.md to match what the
data actually shows.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Literate passes '#hide' through to the markdown, but Documenter still renders
the value of the block's last expression, so Random.seed!(42) was printing a
stray Random.TaskLocalRNG(). Terminated with a semicolon and a hidden nothing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ts chance level

Two small API additions, both prompted by questions that turned out to have
teeth.

1. Bundling is m-way, not pairwise. For BinaryHV, BipolarHV, RealHV and FHRR the
   bundling rule depends on how many inputs there are, so bundling is not
   associative. Chained 'x + y + z' was already safe (Julia parses a chain of +
   as one variadic call, which reaches bundle with all three), but 'sum(hvs)',
   'reduce(+, hvs)' and '(x + y) + z' fold pairwise and bundle a bundle. Measured
   on five hypervectors, the folded result scores 0.08/0.05/0.14/0.26/0.50
   against its inputs where a true bundle scores ~0.37 for all five: the defining
   property of bundling is lost. 'sum' now bundles the whole collection; the
   remaining pairwise spellings are documented as wrong in the bundle docstring.

2. A similarity score is uninterpretable without knowing the metric: 0.35 means
   'unrelated' under Jaccard (BinaryHV, GradedHV) and 'clearly related' under
   cosine. Adds similaritymetric(HV) and, more usefully, chancesimilarity(HV) —
   the score of an unrelated pair, which is the number you actually need. Both
   are locked by a test asserting the documented baseline matches what random
   hypervectors really score, and the encoding tutorial now teaches reading a
   similarity against its chance level.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Runic-formats the two new tutorials; every other changed .jl was already
  conformant, so the style CI should be clean.
- Aligns the colours tutorial's advice on standardising with what the iris
  tutorial actually measures (it is not an automatic win).
- Stops tracking docs/src/examples/*.md. These are Literate output regenerated
  by docs/make.jl on every build, and they accounted for a third of this
  branch's diff (1431 of 4383 lines) as machine-generated duplicates of the .jl
  sources. Verified the docs build from a clean state with no .md present.
- Gitignores .claude/ (local dev config).
- Syncs CLAUDE.md and TODO.md: the new similarity traits, the m-way bundling
  rule, the tutorial list, and a stale entry still advertising the removed
  level/encodelevel/decodelevel/convertlevel family.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@cvigilv

cvigilv commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

I'll review this one as it supersedes everything else; give me the afternoon to do this properly, as it's a big change driven by Claude.

@MichielStock

Copy link
Copy Markdown
Collaborator Author

We can discuss it later today

MichielStock and others added 3 commits July 27, 2026 15:54
Audited the documentation I drafted using the ai-writing-tells skill, with the
project's own prose (the tutorials I did not write) as the style baseline.

Layers 0-1 were clean: no chatbot artifacts, no citation problems. Era-tagged
vocabulary: no hits anywhere. Copula density sits inside the project's own range
(1.2-3.4 per 100 words), so there was no copulative avoidance to undo.

Fixed:
- index.md opened with a four-item list of virtues ('fast, robust to noise,
  remarkably data-efficient, and simple enough...') plus an unsupported
  comparison to deep learning. Replaced with the concrete reason the paradigm is
  robust and fast, and attributed the broader claim to the review paper.
- 'That is the entire/whole X' had become a verbal tic, appearing four times
  across three files. Reworded two of them.
- Literal em dashes: the project uses '--' and never a literal em dash, in any
  file, by any author. My additions had introduced them, all space-surrounded.
  Converted in index.md, inference.jl and my section of encode.jl. The stateful
  encoder docstrings that arrived with the encoders branch are left alone.
- Trimmed two rhetorical intensifiers in colours.jl.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mechanical only: keyword-argument spacing, comprehension indentation, and one
trailing space.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@cvigilv

cvigilv commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

This looks good; I'll merge it and close the other PRs superseded by this one.

Should we publish this as v1.0, or do you want to add something else before that?

@cvigilv

cvigilv commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

@cvigilv
cvigilv merged commit c4faeec into main Jul 29, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants